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From  the  Director's  chair 


E  R  D  C 


Major  Shared  Resource  Center 


The  cover  of  this  newsletter  highlights  new  leadership  and  new  tech¬ 
nology.  The  Department  of  Defense  (DoD)  is  in  good  company  with  the 
Bush/Cheney  team;  General  Flowers,  the  new  Chief,  Corps  of  Engineers 
(COE),  has  a  vibrant  new  vision  for  the  COE  and  promotes  a  “Just  Do  It” 
philosophy;  and  the  new  Director  of  the  Engineer  Research  and  Develop¬ 
ment  Center  (ERDC),  Dr.  James  Houston,  has  a  strong  commitment  to 
high-performance  computing  (HPC).  Cray  Henry  has  been  the  Director, 
High  Performance  Computing  Modernization  Program  (HPCMP),  for 
about  a  year  and  a  half.  More  recent  additions  and  new  assignments  to 
the  High  Performance  Computing  Modernization  Office  (HPCMO)  are 
Dr.  Larry  Davis  as  the  Deputy  Director,  Dr.  Erank  Mello,  Chief  Scientist, 
and  Dr.  Leslie  Perkins,  Programming  Environment  and  Training  (PET) 
and  Common  High  Performance  Computing  Software  Support  Initiative 
(CHSSI)  Lead.  This  new  team  brings  diverse  talent  with  management 
experience,  historical  HPCMP  knowledge,  technical  expertise,  and  computational  expertise  to  the 
HPCMP  leadership  roles.  With  this  strong  cadre  of  new  faces  in  high  places,  we  are  in  a  good  position 
to  embrace  future  challenges. 

On  the  technology  side,  I  want  to  use  the  rest  of  this  space  to  highlight  a  technology  that  has  been  in 
the  minds  of  HPC  people  for  over  two  decades — grid  computing.  It  all  started  with  the  interconnection 
of  computers  with  “networks.”  In  the  early  days,  DECnet  was  a  leader.  Anybody  remember  EAT? 
Although  DECnet  is  still  around,  transport  control  protocol/Intemet  protocol  (TCP/IP)  “stacks”  (I  always 
think  of  pancakes  when  I  hear  this)  eventually  provided  a  standards-based  foundation  for  inter-networking. 
Yes,  some  distractions  occurred  along  the  way;  but  when  the  going  got  tough,  TCP/IP  always  surfaced. 
The  big  thrust  now  is  “grids.”  This  is  more  commonly  known  as  metacomputing,  but  the  community 
quickly  learned  that  metacomputing  meant  too  many  things  to  too  many  people.  A  technology  that  is 
manageable  is  grid  computing.  Initially,  this  was  termed  “computational  grids,”  but  I  shy  away  from  this, 
as  it  steps  on  the  turf  of  the  computational  scientists  who  construct  computational  grids  for  numerical 
solutions.  Hence,  we  move  forward  with  “grid  computing.”  What  is  it?  We  all  know  that  most  of  our 
computers  are  physically  connected  together  with  networks.  Grid  computing  is  layers  of  functionality  on 
top  of  the  network  that  provide  the  end  user  the  ability  to  couple  resources  together  to  address  a  problem 
or  system  of  problems.  Computers  are  the  most  common  resources,  but  devices  like  a  network-attached 
microscope  or  a  measuring  instrument  can  also  contribute. 

The  HPCMP  is  in  the  early  stages  of  building  a  computational  grid.  This  requires  a  “melding”  of  the 
infrastructure  that  involves  not  only  the  hardware  and  software  technologies  but  the  policies,  operating 
procedures,  and  people.  The  whole  community  must  begin  to  think  about  grid  computing  and  what  their 
role  is.  Configuration  control,  security,  user  services,  applications,  user  interfaces,  and  accounting  to 
mention  just  a  few  are  all  impacted.  Most  importantly,  the  user  will  be  impacted.  The  computing  envi¬ 
ronment  will  become  more  complex  and  will  offer  a  capability  far  above  what  is  possible  with  today’s 
infrastructure.  Grid  computing  will  be  an  evolutionary  capability.  In  the  future,  grids  will  be  intercon¬ 
nected  with  other  grids.  You  will  someday  be  able  to  “borrow”  a  telescope  from  the  National  Aeronautics 
and  Space  Administration  (NASA)  for  a  few  hours.  The  possibilities  are  endless. 

We  have  great  new  leadership  and  great  new  technologies.  Look  for  the  two  working  together  to 
change  the  way  we  do  HPC. 


Bradley  M.  Comes 
Director,  ERDC  MSRC 
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announcements 


Rebecca  Fahey  Accepts  Position  as  Director 
of  User  Services 

Rebecca  Fahey  recently  accepted  the  position  of  Director  of  User  Services.  In  this 
position,  she  will  oversee  the  Customer  Assistance  Center,  Applications  Analysts, 
and  Database  Group. 

Rebecca  brings  a  significant  amount  of  technical  expertise  to  the  position.  She 
^  ■  transferred  to  the  position  from  the  Computational  Migration  Group  at  the  ERDC 

Major  Shared  Resource  Center  (MSRC),  where  she  assisted  users  in  the  development, 
migration,  and  optimization  of  parallel  codes.  Prior  to  this  position,  Ms.  Fahey  taught  in  the  University  of 
Kentucky  system,  where  she  was  involved  in  several  projects  designed  to  integrate  more  technology  into 
the  classroom. 

Rebecca’s  technical  expertise  will  blend  nicely  with  the  activities  of  User  Services.  Rebecca’s  current 
goal  is  to  develop  the  technical  skills  of  the  Customer  Assistance  Specialists  to  be  more  in  tune  with  the 
needs  of  the  user  community  through  a  new  support  model  that  puts  much  more  emphasis  on  direct  front¬ 
line  support  versus  managing  customer  calls  through  a  hierarchical  tree  of  expertise. 

Mark  Green  Joins  the  ERDC  MSRC 

Mark  Green  joined  the  ERDC  MSRC  in  October  2000.  Mark’s  focus  is  serving  as  the 
MSRC’s  POC  for  networking  and  mass  storage  systems. 

Mark  has  worked  at  ERDC  for  22  years.  He  began  in  1979  as  a  student  trainee  and  then, 
upon  graduation,  joined  the  professional  staff  in  1982.  Prior  to  joining  the  MSRC  team, 
he  served  as  a  research  civil  engineer  in  the  Geodynamics  and  Explosion  Effects  Division 
of  the  ERDC  Geotechnical  and  Structures  Laboratory,  where  he  conducted  numerous  investigations  to 
characterize  the  response  of  geologic  materials  subjected  to  explosive  loadings  and  to  understand  the 
effects  of  projectile  penetration  into  geologic  targets.  He  has  authored  and  coauthored  20  technical 
reports  and  papers  and  is  a  registered  professional  engineer  in  the  State  of  Mississippi.  Mark  graduated 
from  Mississippi  State  University  with  B.S.  and  M.S.  degrees  in  civil  engineering  and  completed  1  year 
of  advanced  study  in  geotechnical  engineering  at  the  University  of  Colorado. 


New  Lead  on  Grid  Computing  Deveiopment  and 
Systems  Operation 

Greg  Rottman  joined  the  ERDC  MSRC  as  a  civil  engineer  in  June  2000.  He 
has  worked  4  years  in  engineering  and  has  had  13  years  of  experience  in  information 
technology  and  5  years  in  project  management.  Greg  supports  systems  operations 
and  serves  as  the  lead  on  developing  grid  computing  capabilities  throughout  the  HPCMP. 

After  graduating  from  Texas  Tech  University  with  a  B.S.  degree  in  agricultural  engineer¬ 
ing,  Greg  worked  as  a  hydraulic  engineer  for  the  U.S.  Army  Corps  of  Engineers  in  Eort  Worth,  Texas.  In 
1981,  he  accepted  a  position  as  a  civil  engineer  in  the  Automated  Data  Processing  Center  (ADPC)  in  Port 
Worth  as  a  liaison  between  the  ADPC  and  the  Engineering  Division.  He  went  to  work  for  Tektronix,  Inc., 
Graphics  Workstation  Division,  in  1984  as  a  systems  analyst.  After  7  years,  Greg  returned  to  Government 
service  at  the  ERDC  Information  Technology  Laboratory  (ITL),  where  he  worked  in  Customer  Assistance 
and  served  as  project  engineer  and  project  manager  for  the  Electronic  Information  Exchange  Project.  He 
then  worked  for  5  years  in  the  Programs  and  Project  Management  Division  at  the  U.S.  Army  Engineer 
District,  Vicksburg,  before  returning  to  the  ERDC  to  work  in  the  MSRC. 
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Deputy  Program  Manager  Named 

In  August  2000,  John  Mauldin  joined  the  Computer  Sciences  Corporation  (CSC)  and 
assumed  the  responsibilities  of  Deputy  Program  Manager  for  the  MSRC  integration 
contract.  John  brings  a  wealth  of  technical  knowledge  and  experience  to  the  program 
and  will  provide  much  insight. 

John  earned  his  B.S.  degree  in  electrical  engineering  and  M.S.  degree  in  computer 
science  from  Mississippi  State  University.  He  began  his  career  as  an  electrical  engineer 
with  RCA  Missile  and  Surface  Radar  in  Moorestown,  New  Jersey,  where  he  developed 
automated  testing  software  for  production-environment,  quality  assurance  testing  of  radar 
components.  John  later  spent  1 1  years  with  Nichols  Research  Corporation  in  Huntsville,  Alabama.  There 
he  progressed  from  a  technical  contributor  into  management.  His  previous  experience  includes  working 
on  the  ERDC  and  Aeronautical  Systems  Center  (ASC)  HPC  integration  proposal  team.  Prior  to  coming  to 
Vicksburg,  he  worked  for  Sword  Microsystems  in  Huntsville,  Alabama,  as  the  Chief  Operating  Officer 
and  Vice  President  for  Technology  and  Development.  There,  he  also  served  as  Program  Manager  provid¬ 
ing  a  Web-based  on-line  ordering  system  used  in  business-to-business  transactions  for  a  large  printing 
company,  which  was  successfully  prototyped  and  delivered.  Throughout  his  career,  John  has  also  gained 
experience  in  technical  consulting  in  the  areas  of  eBusiness  strategies,  business  process  analysis,  and  re¬ 
engineering. 

While  in  the  Huntsville  area,  John  took  an  active  role  in  the  North  Alabama  Habitat  for  Humanity  organi¬ 
zation  and  served  as  the  President  of  the  Board  of  Directors  for  6  years.  During  his  tenure  as  President, 
over  25  houses  were  completed. 

Outside  of  the  office,  John  enjoys  spending  time  with  his  wife,  Karen,  and  their  three  daughters,  Anna, 
Becca,  and  Ali. 


Upcoming  Events 

Jackson  State  University  High  Performance 
Computing  Summer  Institute  —  June  4-15,  2001 


SC2001  Conference  —  “Beyond 
Boundaries,”  November  10-16,  2001 
Denver,  Colorado,  SC  Goes  Global 


S62001 


Users  Group  Conference  2001  — 

June  18-21,  2001,  Beau  Rivage  Resort 
Biloxi,  Mississippi 


Current  Trends  in  Computational 
Chemistry  2000 

The  9*  Conference  on  Current  Trends  in  Computational  Chem- 
istry  was  held  in  Vicksburg,  Mississippi,  on  November  3-4,  2000. 

This  symposium  was  organized  by  the  ERDC  MSRC  PET  Program 
partner  Jackson  State  University,  one  of  the  Historically  Black 
Universities  and  Colleges/Minority  Institutions  (HBCU/MI).  The  conference  covered  all  areas  of 
computational  chemistry  as  well  as  quantum  chemistry.  The  local  host  of  the  conference  was  the  ERDC 
headquarters  at  Waterways  Experiment  Station  in  Vicksburg. 


Dr.  Phu  Luong,  ERDC  MSRC,  presented  a  virtual  reality  visualization  of  a  Persian  Gulf  simulation  using 
the  Princeton  Ocean  Model  (POM)  for  the  conference  participants  in  the  Collaboratorium  located  in  ITL. 
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Collaboration/Communication  Tutorial 


Professor  Geoffrey  Fox,  Florida  State  University, 
presents  tutorial  at  the  ERDC  MSRC 


As  part  of  the  ERDC  MSRC  PET  Program, 
Professor  Geoffrey  Pox,  Plorida  State  Univer¬ 
sity,  presented  a  tutorial  at  the  ERDC  MSRC 
on  March  1,  2001.  The  current,  state  of  the  art 
in  collaboration  and  communication  was 
presented.  The  tutorial  included  systems, 
tools,  and  standards  for  distributed  meetings, 
training  and  education,  wireless  communica¬ 
tion  technology,  and  high-quality  digital 
audio  and  video  over  the  Internet.  The  Access 
Grid  project  was  discussed  along  with  the 
complex  requirements  for  group-to-group 
interactions  and  the  creation  of  a  distributed 
virtual  environment. 


Lectures  on  Parallel  Adaptive  Algorithms  and  Software  Frameworks 
for  DoD  Challenge  Class  Applications 


Dr.  Robert  T.  McLay,  a  research  associate  at  the  Texas  Institute  for  Computational  and  Applied  Mathemat¬ 
ics,  University  of  Texas,  Austin,  presented  lectures  at  the  ERDC  MSRC  on  January  10,  2001,  describing 
recent  ideas  related  to  data  structures  and  the  use  of  factory  and  framework  concepts  for  development  and 
implementation  of  adaptive  mesh  finite  simulators 
and  for  handling  multiphysics  applications.  General 
concepts  were  described  and  contrasted  with  the 
“traditional”  approach  of  finite  element  analysis  and 
software  engineering.  The  design  patterns  approach 
was  also  discussed  and  examples  given.  To  illustrate 
several  key  ideas  in  the  adaptive  mesh  refinement 
context,  a  TESTbed  factory  and  representative 
results  were  introduced.  Implications  for  wide 
application  of  these  ideas  were  discussed  and  then 
the  extension  of  this  approach  to  three-dimensional 
Parallel  Einite  Element  Eramework.  An  important 
part  of  a  Eramework  for  DoD  applications  is  the 
ability  to  handle  multiphysics  interoperation.  Also 
considered  was  the  problem  of  interfacing  mesh 
generators  like  CUBIT  and  dynamic  repartitioners  with  Bob  Athow,  ERDC  MSRC  PET  Government  Lead 


like  ZOLTAN. 


Strategic  Planning  2001  Workshop 


A  strategic  planning  workshop  for 
the  ERDC  MSRC  was  held  in 
Vicksburg,  Mississippi,  late  January 
and  early  Eebruary  2001  to  discuss 
goals  and  envision  possible  accom¬ 
plishments  for  the  next  5  years  for 
the  ERDC  MSRC. 
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Alcorn  State  University  students,  March  20,  2001,  Wayne  Jones  (far  left),  ITL, 
and  Dennis  Gilman  (far  right),  ERDC  MSRC 


Brad  Comes  (left),  ERDC  MSRC  Director,  and 
COL  Gary  Royster  (right).  Director,  Intelligence 
Policy,  Office  of  the  Deputy  Chief  of  Staff  for 
Intelligence,  The  Pentagon,  March  9,  2001 


Members  of  the  Command  Staff  Inspection  Team, 
Headquarters,  U.S.  Army  Corps  of  Engineers,with 
Brad  Comes,  ERDC  MSRC  Director,  March  8,  2001 


WARPLEX  participants,  February  6,  2001, 
in  the  Collaboratorium,  ERDC  ITL 


Leadership  Vicksburg  participants,  February  6,  2001, 
Dr.  Lisa  Hubbard  (far  left),  ERDC  Coastal  and 
Hydraulics  Laboratory 


Continued  on  page  34 
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New  Leadership 

for  the  ERDC  MSRC 

by  Rose  J.  Dykes 


Exciting  new  leadership  for  the  ERDC  MSRC  inspires  its  personnel 
to  regard  eaeh  new  challenge  as  an  opportunity  to  eontinue  learning 
and  to  imagine  that  unlimited  aeeomplishments  ean  be  made  with 
the  potential  teamwork  of  its  organization  along  with  its  users  and 
associates.  Competent  and  earing  new  leaders  energize  motivation 
and  offer  encouragement  and  support. 


New  leadership  for  the  ERDC  MSRC  begins  with 
the  President  of  the  United  States.  Although  the 
President  is  far  removed  from  the  MSRC,  the  top 
office  in  the  country  has  a  definite  impact.  George 
W.  Bush  was  elected  the  43^^  President  of  the 
United  States  and  proposes  to  strengthen  the 
military  with  “better  pay,  better  planning,  and 
better  equipment.”  New  Vice  President  Richard  B. 
Cheney  earned  the  respect  of  Americans  while 
serving  as  Secretary  of  Defense  with  his  direction 
of  two  large  military  campaigns — Operation  Just 
Cause  in  Panama  and  Operation  Desert  Storm  in 
the  Middle  East.  He  served  three  Presidents  and 
as  an  elected  official  before  his  election  as  Vice 
President. 

The  new  Chief  of  Engineers  at  Headquarters,  U.S. 
Army  Corps  of  Engineers,  LTG  Robert  B.  Elowers, 
shares  a  consistent  vision  with  that  of  ERDC  and 
the  ERDC  MSRC.  His  visits  to  the  ERDC  head¬ 
quarters  in  Vicksburg,  Mississippi,  and  personal 
messages  reinforce  the  ERDC  vision  and  strategy. 
He  expressed  his  philosophy  in  a  video  message 
to  all  ERDC  employees  saying  “first  understand, 
then  seek  win-win  solutions.”  A  “permission  slip” 
signed  by  General  Elowers  was  presented  to  each 
employee  following  the  video  presentation. 

Less  than  a  year  ago.  Dr.  James  R.  Houston 
became  the  first  Director  of  ERDC.  The  U.S. 


Army  Engineer  Waterways  Experiment  Station  (WES) 
is  now  the  site  of  the  ERDC  command  headquarters. 
Dr.  Houston  expressed  his  support  of  HPC  as  follows: 
“High  performance  computing  is  very  important  to  the 
Corps  of  Engineers,  the  Army,  and  the  DoD  to  ensure 
that  the  U.S.  warfighter  has  the  advantage  on  the  battle¬ 
field  of  the  future.  The  ERDC  MSRC,  located  in  the 
ERDC  Information  Technology  Laboratory,  plays  a 
major  role  in  helping  us  provide  this  advantage.” 


Permission  Slip 

Ask  yourself: 

US  Army  Corps 


of  Engineers 


1.  is  it  good  for  my  customer? 

2.  is  it  legal  and  ethical? 

3.  Is  it  something  I  am  willing  to  be  accountable  for? 
If  so,  don't  ask  for  permission.  You/elf^ady  have  it. 

Just  do  it! 


The  DoD  High  Performance  Computing  Modern¬ 
ization  Program  (HPCMP),  with  Cray  Henry  as  its 
Director,  announced  a  new  Deputy  Director  and  a 
new  Chief  Scientist.  Dr.  Larry  P.  Davis  in  his  new 
position  as  Deputy  Director  of  the  HPCMP  will 
assist  Mr.  Henry  in  managing  all  Program  activities. 
He  will  also  continue  to  be  responsible  for  the 
Program’s  requirements  and  resource  allocation 
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activities.  This  essential  process  identifies,  collects, 
and  validates  DoD’s  high-performance  computing 
requirements  and  then  provides  a  mechanism  for  the 
Services  and  Defense  Agencies  to  prioritize  those 
requirements  in  the  form  of  allocations  on  the 
Program’s  HPC  systems.  Included  in  the  effort  is 
management  of  the  set  of  DoD  Challenge  Projects, 
which  represent  the  largest  and  highest  priority 
computational  work  done  within  DoD’s  science  and 
technology  and  test  and  evaluation  communities. 

Dr.  Davis  is  a  research  staff  member  at  the  Institute  for 
Defense  Analyses.  He  has  been  continuously  involved 
with  the  DoD  HPCMP  from  its  inception,  originally 
representing  the  Air  Force  on  the  HPC  Working  Group 
that  founded  the  Program.  During  a  21 -year  Air  Force 
career  that  concluded  in  1994,  Dr.  Davis  performed 
research  at  several  Air  Force  laboratories,  taught 
chemistry  at  the  Air  Force  Academy,  managed  the 
Computational  Chemistry  Research  Program  for  the 
Air  Force  Office  of  Scientific  Research  (AFOSR),  and 
served  as  the  Assistant  Chief  Scientist  of  Air  Force 
Systems  Command  (AFSC).  He  became  deeply 
involved  in  Air  Force  HPC  activities  while  at  AFOSR 
and  AFSC.  Since  1994,  he  has  supported  the  DoD 
HPCMP  in  a  number  of  key  areas. 

Dr.  Davis  holds  a  Ph.D.  in  physical  chemistry  from 
Louisiana  State  University,  an  M.S.  in  systems  man¬ 
agement  from  the  University  of  Southern  California, 
and  a  B.S.  in  chemical  physics  from  Louisiana  State 
University.  He  is  a  member  of  the  American  Chemical 
Society  and  has  published  approximately  20  articles  in 
refereed  scientific  journals. 

In  his  new  position  as  Chief  Scientist  with  the  HPCMP, 
Dr.  Frank  Mello  is  responsible  for  coordinating  HPC 
technical  issues  between  HPC  users  and  the  Program’s 
HPC  initiatives.  Dr.  Mello  will  investigate  optimum 
mapping  of  DoD  HPC  users’  applications  to  available 
HPC  architectures.  He  will  examine  how  efficiently 
these  applications  are  processed  on  different  architec¬ 
tures  and  recommend  the  appropriate  mix  of  HPC 
capabilities  to  support  the  DoD  science  and  technology 
and  test  and  evaluation  missions. 

Dr.  Mello,  a  Principal  Member  of  the  technical  staff  at 
the  Department  of  Energy’s  (DoE)  Sandia  National 
Laboratories  joins  the  HPCMP  from  the  Office  of  the 
Director  of  Operational  Test  and  Evaluation  (DOT&E). 


Within  the  Live  Fire  Testing  Branch,  his  duties 
included  oversight  of  the  Live  Fire  Test  and 
Evaluation  (LFT&E)  of  ballistic  missile  defense 
acquisition  programs,  coordinating  the  LFT&E 
sponsorship  of  the  Target  Interaction  Lethality  and 
Vulnerability  Working  Group  and  interfacing 
between  the  DoE  laboratories  and  the  DOT&E. 

He  represented  DOT&E  and  LFT&E  on  several 
Modeling  and  Simulation  working  groups  dealing 
with  Simulation  Based  Acquisition,  the  Simula¬ 
tion  Test  and  Evaluation  Process,  the  Joint  models 
(JMASS,  JSIMS,  JWARS),  and  HPC.  Dr.  Mello  has 
authored  numerous  publications  on  computational 
mechanics. 

Dr.  Leslie  Perkins  is  the  new  Project  Manager  for 
the  CHSSI/PET  activities.  Dr.  Perkins  joins  the 
HPCMP  from  the  Air  Force  Research  Laboratory’s 
(AFRL)  Propulsion  Directorate  at  Edwards  Air 
Force  Base,  California.  As  Assistant  Chief 
Scientist  for  AFRL-Edwards,  her  duties  included 
overseeing  the  basic  research  efforts — totaling 
about  $2.4  million — conducted  at  the  Edwards’ 
research  site.  She  was  also  responsible  for  the 
final  technical  review  of  all  in-house  and  contract 
programs.  She  supervised  1 1  Air  Force  Palace 
Knight  Program  participants  and  served  as 
Program  Element  Monitor  for  basic  research 
activities.  Her  experience  includes  administering 
the  National  Research  Council  and  AFOSR 
Summer  Research  Programs.  Over  the  last  3  years. 
Dr.  Perkins  has  served  as  the  Computational 
Chemistry  and  Materials  Science  Computational 
Technology  Area  Lead  for  the  DoD. 

Dr.  Perkins  holds  a  Ph.D.  in  physical  chemistry 
from  Iowa  State  University  and  a  B.S.  degree  in 
chemistry  from  the  University  of  New  Orleans. 
The  ERDC  MSRC  looks  forward  to  Dr.  Perkins 
leading  the  HPCMP  into  the  future  with  the 
follow-on  PET  Program  and  the  multidisciplinary 
CHSSI  Program. 

The  ERDC  MSRC  is  pleased  with  the  new 
leadership  in  place  that  will  help  to  ensure  that  its 
mission  to  deliver  HPC  leadership,  service, 
education,  and  technical  expertise  to  achieve 
research  and  engineering  objectives  vital  to  the 
Nation  will  be  accomplished. 
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by  Dauid  Stinson 


Each  year  the  HPC  modernization  Program  [HPCmP]  participates 
in  SCxy,  a  large  conference  luhere  computer  hardmare  and 
softmare  vendors  shorn  off  their  mares,  and  universities  and 
research  organizations  present  projects  that  they  have  m 
the  morhs. 


During  the  meeh  of  Dovember  4-I0,  Dallas,  Texas,  served 
as  the  host  site  for  SC2DDD.  SC2DD0  is  sponsored  by  the 
Institute  of  electrical  and  Electronics  Engineers  Eomputer 
Society  and  the  Association  for  Eomputmg  machinery’s  j 
Special  Interest  Group  on  Eomputer  Drchitecture.  The  DoD  , 
HPEmP  has  participated  m  the  conference  each  year 
since  I99A. 

Dem  features  added  to  this  year's  conference  included 
Denture  Uillage,  eSEape  2DDD,  a  reconfigured  HPE  Games, 
Eo-Ehaired  by  DDUD  mSRE's  Eleanor  Schroeder,  and  a  nem 
netmorhmg  amard.  Denture  Dillage  shomcased  a  collection 
of  entrepreneurial  information  technology  companies, 
all  creating  nem  products  to  build  the  infrastructure  of 
tomorrom.  The  eSEape  2000  thrust  demonstrated  the  ability 
to  “escape "  from  today's  technological  boundaries  and  to  connect 
and  compute  anymhere.  messrs.  Stephen  Jones  and  John  UJest, 
ERDE  mSRE,  served  as  Eo-Ehairs  for  this  event.  The  SEinet  2000 
netmorh  shomcased  leading-edge  technology  throughout  the 
conference.  Dr.  Louis  Turcotte,  formerly  of  the  ERDE  mSRE, 
served  as  the  overall  Eonference  Ehair.  After  presentations 
of  tutorials  and  the  education  program  over  the  meehend  and 
through  monday  morning,  the  convention  center  floor  mas 
opened  to  the  public  monday  evening  mith  a  Gala  Opening  Reception 
The  meeh  mas  laced  mith  talks  from  invited  speakers,  presenta¬ 
tions  of  technical  papers  and  amards,  panel  discussions,  and 
Dirds-of-a-Feather  sessions. 


The  DoD  HPGmP  mas  mell  represented.  In  addition  to  the  many 
HPGrnP  persons  serving  m  key  leadership  and  planning  roles  m 
the  conference,  HPE  modernization  Program  participants 
presented  papers  and  posters  [referred  to  as  Research  Gems 
this  year]. 


Each  year,  one  of  the  HPEmP  mSREs  is  responsible  for  hosting 
a  booth  at  the  conference.  This  year,  the  responsibility  rested! 
mith  the  ERDE  mSRE.  Although  many  hours  of  hard  mork  mere 
spent  preparing  for  the  booth,  those  participating  m  the  ! 

preparation  found  it  a  great  experience.  Only  one  mSRE  is 
responsible  each  year  for  organizing  the  DoD  booth,  but  the  other 
mSREs  and  Distributed  Eenters  provide  input  m  the  may  of 
graphics,  demonstrations,  and  booth  morkers.  The  experience  for 
all  mas  great-the  leading-edge  technology  exhibited  mas 
extraordinary. 


SC2000  DoD  HPCmP  Booth  Setup 
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David  Stinson  (ieft),  DoD  HPCMP  Booth  Chairman, 
and  Charies  Ray  (right),  ERDC  MSRC,  pose  for 
a  picture  as  the  booth  setup  is  aimost  compiete 


(Left  to  right): 
Chesiey  Cuicchi, 
Keiiy  Lanier, 
Jeanie  McDonald, 
and  Dean  Hampton, 
all  of  the  ERDC 
MSRC,  practice 
with  the  information 
kiosk  before  the 
reception  begins 
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Brad  Comes,  ERDC  MSRC  Director, 
tests  the  information  kiosk 


Mike  Moore,  ASC  MSRC,  Eieanor  Schroeder, 
Navai  Oceanographic  Office  (NAVO)  MSRC, 
and  Dean  Hampton,  ERDC  MSRC 


NAVO  MSRC 
group  at 
HPCMP  booth 
(ieft  to  right): 
Bob  Knesei, 
Pete  Gruzinskas, 
Eieanor  Schroeder, 
Christine  Cuicchi, 
and  Steve  Adamec 


(Left  to  right):  Dr.  Louis  Turcotte,  Conference  Chair, 
Stephen  Jones,  eSCape  2000  Co-Chair,  and 
Dr.  Dennis  Duke,  Venture  Viiiage  Vice  Chair 


David  Stinson,  DoD  HPCMP  Booth  Chairman,  gives 
instructions  to  the  ERDC  MSRC  Outreach  Team 
members  before  the  Gaia  Opening  Reception 


Pete  Gruzinskas,  NAVO  MSRC,  Frank  Cameron  and 
Rita  wakey,  Navai  Air  Warfare  Center, 
and  Steve  Schneiier,  Navai 
Undersea  Warfare  Center 
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Shown  visiting  with  Dr.  Louis  Turcotte 
(center),  Conference  Chair,  in  the  Research 
Gems/eSCape  2000  area  of  the  conference 
are  (ieft  to  right)  Charies  Ray,  Rose  Dykes, 
Chesiey  Cuicchi,  Keiiy  Lanier,  Jeanie 
McDonaid,  and  Dr.  Phu  Luong,  aii  of  the 
ERDC  MSRC 


An  eSCape  2000/Research  Gems  shirt  is  donned  by 

someone  speaking 
with  Stephen  Jones, 
eSCape  2000 
Co-Chairman 


SCinet  2000 
estabiished 
fiexibie,  wide- 

area  connections  to  the  show  fioor,  instaiiing  and  operating  more  than 
40  miies  of  fiber  optics  throughout  the  conference  areas 


Entrance  to  Venture  Viiiage 


Entrance  to  Research  Gems/ 
eSCape  2000 


Entrance  to  SC2000  Convention  fioor 


Dr.  Phu  Luong,  ERDC  MSRC,  participates  in  the  Research  Gems 
sessions  with  his  poster  presentation  entitied  “Advantages  of 
Muitibiock  Curviiinear  Grid  and  Duai-Levei  Paraiieiisms 
in  Ocean  Circuiation  Modeiing” 


Rebecca  Fahey,  ERDC  MSRC,  and  her  award-winning 
poster  entitied  “Steady-State  Spectrai  Wave 
Modei  (STWAVE)—A  Case  Study 
in  Duai  Levei  Paraiieiisms” 
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Pete  Gruzinskas,  NAVO  MSRC,  demonstrates  HPC 
research  projects  to  a  young  visitor 


Susan  Pfeiffer-Vega,  HPC  Modernization  Office, 
at  the  HPCMP  booth  greeter’s  tabie 


Professor  Shahrouz  Aiiabadi,  dark  Atianta  University, 
views  HPC  research  projects  at  DoD  HPCMP  booth 


Jeff  Graham  and  Hank  Laughiin,  ASC  MSRC, 
aiong  with  Dean  Hampton,  ERDC  MSRC 


Cathy  McDonaid,  HPC  Modernization  Office,  and  Eieanor 
Schroeder,  NAVO  MSRC,  man  booth  as  Steve  Adamec 
(ieft),  NAVO  MSRC  Director,  and  Dr.  George  Heburn 
(right),  NAVO  MSRC,  visit 


Jay  Ciiburn,  ERDC  MSRC,  visits  at  booth  with  Virginia 
Bedford,  Arctic  Region  Supercomputing  Center 


I  *i 

'  *  M 

vBhB 

ft  T^lR 

HBIv^bR.  — ij 

cj^"'  jBI _ 

■=^ 

B  MHI  U  ^a 

IL».JI 

ifi 

JKUJm 

SSiC 

Ntf 

hp^?M 

L—  .-^  ~  .: 

iiF  ;■ 

;] 

;i 

i£^  J 

EmSIl^  J 

c 

m 

■m 

Ml 

i:', 

T  •■■' 

ffe 

!• 

■?“ 

iilliilL 

In 

m 

m 

-T”^^ 

1 

“ 

* 

■SL 

li 

— 

E3BHHUW 

|TO 

li 

? 

■■If  ~ 

■■ 

HB  99- 

H'— 

ffjH 

ImTjQ 

1  I 

^fflHjTll 

14 


Charles  Ray 
ready  with 
the  truck  for 
leaving  Dallas 


Brad  Comes,  ERDC  MSRC  Director,  assists  in  tear  down 
of  booth  as  Pete  Gruzinskas,  NAVO  MSRC,  and  Charles 
Ray,  ERDC  MSRC,  watch  with  some  concern 


Charles  Ray,  ERDC  MSRC,  Frank  Lavoto, 
NAVO  MSRC,  and  Dennis  Gilman, 
ERDC  MSRC 


SC2000  DoD  HPCMP  Booth 


ERDC  MSRC  group  smiles  as  everything  is  packed  at  the  end  of  SC2000 
Front  row  (left  to  right):  Rose  Dykes,  Dean  Hampton,  Jeanie  McDonald,  Kelly  Lanier,  David  Longmire 
Second  row  (left  to  right):  Dana  Allen,  David  Stinson,  Chesley  Cuicchi,  Brad  Comes 


In  back:  Charles  Ray 
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Damage  Simulations  in  Hard  and  Deeply  Buried 
Targets  as  a  Result  of  Blast  and  Shock  Loading 

Dr.  G.  W.  McMahon,  B.  J.  Armstrong,  Jr,  and  C.  E.  Joachim 


The  counterproliferation  of  weapons  of  mass 
destruction  (WMD)  continues  to  remain  one  of 
the  Nation’s  highest  defense  priorities.  The  defeat 
of  systems  within  hard  and  deeply  buried  targets, 
namely  tunnels,  presents  the  greatest  challenge. 
When  a  weapon  detonates  in  a  confined  area,  the 
airblast  effects  are  much  more  severe  than  those 
from  a  free-air  detonation.  The  size  and  complex¬ 
ity  of  numerical  models  required  to  assess  func¬ 
tional  damage  to  a  facility  require  HPC  resources. 

The  geometries  of  hard  and  deeply  buried  targets 
in  rock  range  from  long  straight  tunnels  to  com¬ 
plex  geometries  with  multiple  levels.  Substantial 
HPC  resources  are  required  for  an  accurate  numeri¬ 
cal  model  representation  of  the  complex  three- 
dimensional  (3-D)  geometry  and  functional 
layout,  the  interaction  of  the  blast  wave  with 
mission-critical  and  support  equipment,  and  the 
response  of  the  equipment  to  failure. 

Computer  Codes  used  in  Numerical  Modeling: 
Blast  environments  were  simulated  using  the 
scalable  hydrodynamic  code  CTH.  The  equip¬ 
ment  response  was  simulated  with  a  parallel 
version  of  DYNA  (ParaDyn),  a  finite  element 
code.  The  calculations  were  conducted  on  the 
ERDC  MSRC  Cray  T3E,  IBM  SP,  and  SGI  Origin 
2000.  The  calculations  involve  modeling  high- 
explosive  initiation  and  detonation  effects,  large 
deformations,  and  damage.  The  problems  are  also 
3-D  and  involve  complex  interactions  with  objects 
in  the  blast  flow.  CTH  [1]  and  ParaDyn  [2]  are 
multidimensional  scalable 
computer  codes  widely 
used  in  the  defense 
research  and  development 
community  to  model 
shock  physics  and  large 
deformation  problems. 

CTH  is  a  finite  volume- 
based,  explicit  time 
integration  method  and 
uses  Eulerian  formula¬ 
tions.  ParaDyn  is  a  rate- 
based,  finite-element. 


explicit  time  integration  method  and  uses 
Lagrangian  formulations. 

Airblast  Simulation:  Ideally,  the  entire 
calculation — from  the  initiation  of  a  charge,  blast 
propagation,  interaction  of  the  blast  wave  with  the 
equipment,  and  equipment  response — should  be 
conducted  in  three  dimensions  as  a  single,  coupled 
calculation.  Limits  to  computational  resources 
require  that  approximations  be  made  to  the 
geometry  in  order  to  reduce  the  calculation  size. 
Since  the  environment  calculation  must  be 
decoupled  from  the  loading  and  equipment 
response,  only  limited  aspects  of  the  problem  can 
be  simulated  in  a  detailed  3-D  grid.  Eor  the  tunnel 
geometry  in  Eigure  1,  the  differences  in  problem 
size  and  the  computational  resources  required  for 
these  two  approaches  are  given  in  Table  1.  If  the 
tunnel  geometry  is  represented  as  a  planar  2-D 
structure,  the  locahzed  charge  has  to  be  distributed 
so  that  the  mass  of  the  distributed  charge  equals 
the  mass  of  the  concentrated  charge  (Eigure  2). 
The  2-D  calculation  results  for  this  tunnel  are 
compared  with  results  from  a  3-D  calculation  in 
Eigure  3.  The  3-D  calculation  waveforms  are 
compared  with  experimental  data  in  Eigure  4. 
Eigures  5  and  6  show  the  blast  wave  propagation 
down  the  tunnel  before  and  after  engulfment  of 
the  storage  tank.  The  reflected  shock  from  the 
back  wall  and  the  multiple  shock  reflections  from 
the  alcove  are  evident  in  the  waveforms. 


Table  1 

Comparison  of  Computafional  Resources 

Tor  2-D  and  3  D  Tunnel  Airblas*  CaiculaUons 

Resource 

2‘D  Calculation 

3*0  Calculation 

Machine 

IBM  SP 

T3e 

Processors 

96 

256 

Code 

Parallel  CTH 

Parallel  CTH 

Grid  Size 

0.8  million  cells 

33  millfon  cells 

Mimmum  Cell  Size 

50.0  mm 

60.0  mm 

Wldtii  of  Grid 

BO  meters 

80  mtiieris 

Kfeight  or  Gnd 

25  meters 

25  meters 

Memory  Si^e 

6.5  MBytes 

12  GBytes 

Ai/erage  Fite  Dump  Size 

1 60  MBytes 

65  CByles 

Run  Time 

1.000  cpu  tiours 

25,000  cpu  hours 
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Figure  1.  Full-scale  tunnel  layout 


Figure  2.  Two-dimensional  planar  model 
of  tunnel 


1.00  10.00  100.00 

Range  Meters 


Figure  3.  Comparison  of  pressure  versus  range 
from  CTH  2-D  and  3-D  calculations 


Figure  4.  Comparison  of  pressure  history 
from  3-D  model  with  test  results 
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Figure  5.  Blast  wave  propagation  from  3-D  calculations 
in  tunnel  before  engulf ment  of  storage  tank 


Figure  6.  Blast  wave  propagation  from  3-D  calculations 
in  tunnel  after  engulf  ment  of  storage  tank 


Equipment  Loading  and  Response:  In  these 
calculations,  the  storage  tank  is  modeled  as  a  solid 
steel  structure.  Assuming  that  the  tank  is  a  solid, 
rigid  structure,  any  modification  of  the  load  on  the 
structure  that  would  result  from  interaction  of  the 
airblast  with  an  actual  deforming  structure  is  lost. 
However,  these  losses  are  considered  small,  as  the 
tank  orientation  does  not  change  significantly 
during  engulfment.  The  loads  calculated  on  the  2- 
and  3-D  solid  structure  are  applied  to  a  3-D  finite 
element  representation  of  the  tank.  Comparisons 
of  test  results  with  calculations  for  the  tank 
response  are  shown  in  Figure  7. 


Discussion  of  Results:  The  discrepancy 
between  the  2-D  numerical  simulation  and 
the  experiment  appears  to  be  the  result  of  the 
planar  approximation  in  which  a  concen¬ 
trated  charge  is  simulated  with  one  that  is 
distributed  across  the  cross  section  of  the 
tunnel.  The  3-D  calculation  of  the  detona¬ 
tion  and  airblast  propagation  reproduced  the 
shock  arrival  and  the  incident  shock  strength 
quite  accurately.  The  arrival  of  the  reflections 
in  the  calculation  is  not  consistent  with  the 
data  and  is  the  result  of  the  reflection  from 
the  back  wall  and  multiple  shock  interactions 
at  the  airblast  gauge  location  at  Station  48. 
The  response  of  the  tank  was  overpredicted; 
but  at  this  level  of  response,  small  variations 
in  the  load  can  lead  to  large  increases  in 
response.  Overall,  the  2-D  planar  approxi¬ 
mations  appear  to  lead  to  unacceptable 
results,  and  problems  of  this  type  will  have 
to  be  addressed  by  3-D  calculations.  Even 
with  the  large  computing  resources  that 
would  be  required,  using  3-D  numerical 
models  is  much  more  cost  effective  than 
conducting  full-scale  experiments.  In  the 
near  future,  the  plans  are  to  validate  coupled 
Eulearian/Lagrangian  codes  against  this 
class  of  problems. 
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Figure  7.  Calculated  (with  loads  from  CTH)  and  experimental  response  of  storage  tank 


Dr.  McMahon  and  Mr.  Joachim  are  research  civil  engineers,  and  Mr.  Armstrong  is  a  research 
physicist  in  the  Geotechnical  and  Structures  Laboratory  at  the  U.S.  Army  Engineer  Research 
and  Development  Center  in  Vicksburg,  Mississippi. 
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Fifth  Annual  ERDC  MSRC  PET  Review 


Robert  F.  Athow 


(Left  to  right):  Brad  Comes,  ERDC  MSRC  Director; 
Bob  Athow,  Government  PET  Lead;  Professor  Shahrouz 
Aiiabadi,  dark  Atianta  University;  John  Mauidin, 
CSC  Deputy  Program  Manager 


(Left  to  right):  Dr.  Jeffery  Hoiland,  Government  CTA  Lead  for 
EQM;  Dr.  Phu  Luong,  EQM  Onsite  Lead;  Professor  Mary 
Wheeier,  University  of  Texas,  Austin;  Professor  Bharat  Soni, 
Mississippi  State  University;  Professor  Joe  Thompson, 
Academic  Team  Lead,  Mississippi  State  University; 
Brad  Comes,  ERDC  MSRC  Director 


February  28,  2001,  saw  the  Fifth  Annual  ERDC 
MSRC  Programming  Environment  and  Training 
(PET)  Review  draw  to  a  close.  The  PET  partners 
presented  a  broad  spectrum  of  timely  reviews  of 
their  support  and  projects  to  the  HPC  community. 
While  torrential  rain  sought  to  dampen  the 
presentations,  the  quality  of  the  products  and 
projects  brightened  the  outside  gloom. 

Surprise  recognition  was  given  to  Professor 
Emeritus  Wayne  Mastin  for  his  dedicated  service 
to  the  ERDC  MSRC  PET  Program.  The  ERDC 
MSRC  Director,  Brad  Comes,  and  the  ERDC 
MSRC  Government  PET  Lead,  Robert  (Bob) 
Athow,  presented  a  special  commendatory  plaque 
to  him. 

The  newest  personnel  to  the  PET  Program  were 
introduced  to  the  assembly:  Dr.  Leslie  Perkins, 
Common  High  Performance  Computing  Software 
Support  Initiative  (CHSSI)/PET  Project  Manager 
for  the  HPCMP;  John  E.  West,  ERDC  MSRC 
Onsite  PET  Director  and  Director  of  Scientific 
Computing;  and  Mr.  Athow.  Brief  introductory 
remarks  were  made  by  each  of  them. 

Environmental  Quality  Modeling  and  Simulation 
(EQM)  activities  were  highlighted  by  Professor 
Mary  F.  Wheeler,  University  of  Texas,  Austin, 
and  included  improving  the  parallel  performance 
of  environmental  quality  models,  full  two- 
dimensional  (2-D)  parallel  implementation  of 
CH3D-Z,  interpolation  and  projection  between 
arbitrary  space/time  grids,  and  reactive  transport 


Front  tabie  (Left  to  right):  Professor  Joe  Thompson,  Academic 
Team  Lead,  Mississippi  State  University;  Doug  Waiker,  CSC 
Program  Manager;  Dr.  Lesiie  Perkins,  CHSSl/PET  Project 
Manager,  HPCMP;  Bob  Athow,  Government  PET  Lead; 
Brad  Comes,  ERDC  MSRC  Director; 
and  John  West,  ERDC  MSRC  PET  Director 


(Left  to  right):  Brad  Comes,  ERDC  MSRC  Director;  Dr.  Leslie 
Perkins,  CHSSl/PET  Project  Manager,  HPCMP; 

Bob  Athow,  Government  PET  Lead 
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schemes  for  water  quality  modeling  on  unstructured 
grids.  A  Climate,  Weather,  and  Ocean  Modeling  and 
Simulation  (CWO)  scientific  visualization  demonstra¬ 
tion  by  Dr.  Phu  Luong,  ERDC  MSRC,  was  particu¬ 
larly  interesting  to  the  attendees.  Professor  Clint 
Dawson,  also  from  the  University  of  Texas,  provided 
insights  to  his  work. 

Specific  EQM  projects  addressing  parallel  implemen¬ 
tation  issues  relating  to  the  groundwater  contaminant 
transport  code  EEMWATER  were  presented  by 
Professors  Tom  McGehee  and  Mike  McAdams  of 
Texas  A&M  University,  Kingsville. 


Professor  Graham  Carey  and  Dr.  David  Littlefield, 

University  of  Texas,  Austin,  and  Dr.  Richard  A. 

Weed,  ERDC  MSRC,  presented  the  Computational 
Structural  Mechanics  (CSM)  core  support  and  project  activities. 
Algorithm  enhancements,  adaptive  mesh  technology  and  mesh 
improvement  for  hypervelocity  impact  and  penetration  analysis, 
discontinuous  Galerkin  EE  and  adaptive  grids,  error  and  feature 
indication  in  interpolation  and  approximation  algorithms,  and 
hierarchical  data  mining  of  very  large  data  sets  for  remote 
visualization  and  manipulation  were  all  presented. 


Commendatory  plaque  was  presented  to  Professor  Emeritus 
Wayne  Mastin  for  his  service  to  the  ERDC  MSRC  PET 
Program.  (Left  to  right):  Bob  Athow,  Brad  Comes, 
Professor  Mastin,  John  West,  Professor  Joe  Thompson 


Professor  Shahrouz  Aliabadi,  Clark  Atlanta 
University,  discusses  parallel  simulation 
of  waves  interacting  with  marine 
vessels  in  motion 


Professor  Bharat  Soni,  Mississippi  State  University,  and  Dr.  Nathan 
Prewitt,  ERDC  MSRC,  presented  the  work  within  the  Computa¬ 
tional  Fluid  Dynamics  (CFD)  core  support  and  project  sphere 
including  grid  assembly  enhancements  for  Chimera  technology, 
JAVA-based  CFD  training  tools,  and  a  library  of  interpolation  and 
approximation  models  (INLiB).  Professor  Shahrouz  Aliabadi, 
Clark  Atlanta  University,  addressed  two  projects,  parallel  simula¬ 
tion  of  wave  interacting  with  marine  vessels  in  motion  and  parallel 
remote  visualization.  He  also  showed  a  one-billion  tetrahedral 
unstructured  element  mesh  application  with  a  sustained  speed  of 
over  230  gigaflops  on  a  Cray  T3E  with  1,088  processing  elements. 


4  i 


(Left  to  right):  Beverly  Bradley  and  Eigoro  Hashimoto, 
Naval  Oceanographic  Office,  shown  with  Kelly  Lanier, 
PET  Training  Coordinator 


(Left  to  right):  Professor  Keith  Bedford  and  Dr.  David 
Welch,  Ohio  State  University,  and  Dr.  Stephen 
Wornom,  CWO  Onsite  Lead 
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Professor  Keith  Bedford  and  Dr.  David  Welsh,  Ohio  State  University,  and 
Dr.  Stephen  Wornom,  CWO  Onsite  Lead,  presented  the  Climate,  Weather, 
and  Ocean  Modeling  and  Simulation  activities,  which  included  application 
of  SWAN  and  the  coupling  of  wave-current- sediment  models  (COMAPS) 
for  nearshore  and  tributary  plume  predictions. 


Collaboratory  actions  were  the  focus  of  the  presentations  by  Dr.  Polly 
Baker,  University  of  Illinois,  in  Scientific  Visualization,  Dr.  Wojtek 
Furmanski,  Syracuse  University,  with  the  FMS  SPEEDES  installation 
and  training,  and  Professor  Geoffrey  C.  Fox,  Florida  State  University, 
communication/collaboration  with  modular  collaborative  environments 


(Left  to  right):  Dr.  Jeffery  Holland, 
Government  CTA  Lead  for  EQM;  Dr.  Phu 
Luong,  EQM  Onsite  Lead;  and  Professor 
Clint  Dawson,  University  of  Texas,  Austin 


All  presenters  focused  on  their  continued  efforts  in  training 
users,  both  onsite  and  remotely.  Professor  Mastin  specifically 
highlighted  the  ERDC  MSRC’s  onsite  training  activities. 

The  summer  intern  program  at  Clark  Atlanta  University  and 
the  classes  with  Jackson  State  University  were  given 
particular  credence. 


Scalable  Parallel  Processing  Tools  presentations  were  made  by  Dr.  Richard 
Hanson,  Rice  University,  Professor  Behrooz  Shirazi,  University  of  Texas, 
Arlington,  and  Dr.  David  Cronk,  University  of  Tennessee. 

Support  for  PARSA  using  fortran  pthreads  and  the  SARA-3D 
structural  technology  acoustics  applications  were  all  of  interest. 


Dr.  Polly  Baker,  University  of  Illinois,  shown 
with  Professors  David  Littlefield  and 
Graham  Carey,  University  of  Texas 


The  overriding  success  of  this  annual  meeting  is  best  illustrated 
by  the  many  formal  and  informal  interactions  and  collaborations 
that  crosscut  all  the  computational  technology  areas.  And  it  was 
fun  and  interesting  too! 


(Left  to  right): 

Dr.  Nathan  Prewitt,  CDF 
Onsite  Lead,  visits  with 
Dr.  Ralph  Noack,  Army 
Research  Laboratory 


(Left  to  right):  Bob  Athow,  Government  PET 
Lead;  Professor  Joe  Thompson,  Academic 
Team  Lead,  Mississippi  State  University; 
John  West,  ERDC  MSRC  PET  Director 


(Left  to  right):  Professors  Jeff  Marquis  and 
Behrooz  Shirazi,  University  of  Texas,  Arlington, 
and  Dr.  Richard  Hanson,  Rice  University 


(Left  to  right):  Dr.  Nathan  Prewitt,  CFD  Onsite  Lead; 
Professor  Bharat  Soni,  Mississippi  State  University; 
Dr.  Ralph  Noack,  Army  Research  Laboratory;  Hugh 
Thornburg,  Aeronautical  Systems  Center  MSRC 
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Shown  visiting  between  presentations  (ieft  to  right): 
Dr.  Jeffery  Hoiland,  Government  CTA  Lead  for 
EQM;  Brad  Comes,  ERDC  MSRC  Director;  and 
Professor  Bharat  Soni,  Mississippi  State  University 


(Left  to  right): 
Professor  Mary 
Wheeier,  University 
of  Texas,  Austin, 
and  Dr.  Fred  Tracy, 
ERDC  Scientific 
Advisor  for  PET 


Professor  Keith  Bedford,  Ohio 
State  University,  discussed  CWO 
core  support — onsite  and  offsite 


(Left  to  right):  Professors  Wojtek 
Furmanski,  Syracuse  University 
and  Geoffrey  Fox, 
Fiorida  State  University 


Professor  Mary  Wheeier, 
University  of  Texas,  and 
Professor  Joe  Thompson, 
Mississippi  State  University 


Professor  Behrooz  Shirazi,  University  of 
Texas,  Ariington,  and  Dr.  Nathan  Prewitt, 
CFD  Onsite  Lead 


Professors  Behrooz  Shirazi  and  Jeff  Marquis, 
University  of  Texas,  Ariington 


Dr.  Robert  Hail,  ERDC  Geotechnical  and 
Structures  Laboratory,  and  Dr.  Rick  Weed, 
CSM  Onsite  Lead 


(Left  to  right):  Dr.  Phu  Luong,  EOM  Onsite  Lead; 
Professor  Clint  Dawson,  Dr.  Lea  Jenkins,  and 
Professor  Mary  Wheeier,  University  of  Texas,  Austin 
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IBM  SP  Switch2 

Jay  Cliburn 


Recently,  the  ERDC  MSRC  upgraded  the  inter¬ 
connection  fabric  of  the  IBM  Power3  symmetric 
multiprocessor  (SMP)  machine  called  “Cobalt.” 
The  new  interconnection  switch  is  known  as  the 
SP  Switch2.  Cobalt  consists  of  64  nodes  contain¬ 
ing  eight  processors  each,  for  a  total  of  512 
central  processor  units  (CPUs).  Each  node  runs 
its  own  copy  of  the  AIX  operating  system. 

The  interconnection  fabric  that  binds  the  indi¬ 
vidual  nodes  into  a  cooperative  whole  is  called  the 
SP  Switch,  or  simply  “the  switch.”  Whenever  a 
parallel  process  needs  to  pass  information  to  a 
cooperating  process  on  another  node,  the  informa¬ 
tion  is  passed  over  the  switch.  The  specific 
mechanism  employed  by  each  process  to  effect 
the  information  transfer  is  called  a  switch  window. 
Switch  windows  are  assigned  to  processes  at  job 
start-up  time. 


Prior  to  the  switch  upgrade,  any  given  Cobalt 
node  could  support  a  maximum  of  four  switch 
windows,  thus  imposing  an  upper  limit  of  four 
simultaneous  message-passing  interface  (MPI)  or 
serial  processes,  even  though  each  node  contains 
eight  CPUs.  The  switch  window  deficit  was 
eliminated  with  the  installation  of  the  SP  Switch2. 
There  are  now  sixteen  switch  windows  available 
per  node;  one  for  each  of  the  eight  CPUs  plus  a 
surplus  of  eight  windows  for  future  growth. 

The  SP  Switch2  enables  full  utilization  of  the 
installed  8-way  nodes,  higher  job  densities,  and 
more  efficient  placement  of  jobs  on  the  ERDC 
MSRC  Powers  SMP  machine,  resulting  in  lower 
overall  job  queue  wait  times  for  user  jobs.  The 
ERDC  MSRC  is  excited  to  offer  this  increased 
capability  to  its  customers. 


ERDC  MSRC  Network  Status 

Michael  Nassour 

The  bandwidth  capacity  of  the  Internal  Major 
Shared  Resource  Center  (MSRC)  Network  (IMN) 
has  changed  very  little  since  1996.  The  current 
backbone  technology,  Tiber  Distributed  Data 
Interface  (EDDI),  with  a  theoretical  transfer  rate  of 
100  Mbps  (Megabits  per  second),  experiences 
sustained  utilization  of  more  than  40  percent  with 
peak  utilization  at  or  near  100  percent.  In  con¬ 
trast,  the  network  capability  of  the  computing 
systems  within  the  MSRC,  as  well  as  the  data 
movement  requirements  of  the  user  community, 
has  steadily  increased.  Recent  initiatives  for 
sharing  and  storing  data  have  significantly  in¬ 
creased  the  backbone  infrastructure  capability 
within  the  MSRC. 

The  two  most  noticeable  improvements  to  the 
IMN  were  converting  the  interactive  EDDI- 
accessed  systems  to  ATM  OC-12c  (622  Mbps) 


interfaces  and  implementing  a  Gigabit  Ethernet 
(1,000  Mbps)  backbone  for  internal  data  transfers. 
The  Defense  Research  and  Engineering  Network 
(DREN)  users  should  notice  a  drastic  improve¬ 
ment  in  interactive  connectivity,  as  many  interme¬ 
diate  translation  devices  have  been  removed;  i.e., 
OC-12  from  DREN  to  EDDI  interactive  became  a 
direct  OC-12  “pipe”  from  DREN  to  the  HPC 
system.  The  larger  “pipe”  provides  numerous 
opportunities  for  inter-MSRC  initiatives  such  as 
metacomputing. 

The  upgrades  were  implemented  in  the  most 
transparent  way  possible  to  the  users  by  using  a 
domain  name  system  (DNS)  to  resolve  host 
names.  If  one  used  host  entries  in  local  configura¬ 
tion  files,  changes  to  the  setup  were  required  to 
address  the  new  network  infrastructure. 
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Grid  Computing  Deveiopment 

Greg  Rottman 

Development  of  grid  computing  technology 
continues.  Greg  Rottman  attended  the  Grid 
Forum  meeting  held  in  Boston  on  October  15-18, 
2000.  The  Grid  Forum  is  a  voluntary  organization 
that  helps  create  grid  technologies  and  inter¬ 
operability  while  reducing  duplicated  efforts. 

Greg  also  represents  the  ERDC  MSRC  on  the 
MSRC  Metacomputing  Working  Group  (MCWG). 


The  MCWG  has  representatives  from  across  the 
program  that  work  to  promote  the  development  of 
computational  grid  technology  across  all  the 
centers.  Greg  continues  to  work  closely  with  the 
MCWG  to  develop  a  plan  for  the  initialization  of 
a  project  to  create  a  computational  grid  for  the 
HPCMP. 


New  Origin  3800  System 

Rebecca  Fahey 

The  ERDC  MSRC  is  pleased  to  announce  its 
newest  high-performance  computer,  an  SGI 
Origin  3800  (03K),  which  replaced  an  SGI  Origin 
2800  system  (02K).  The  new  system  was  initially 
four  128-processor  images  that  ultimately  became 
a  single  512-processor  image.  This  new  system 
provides  eight  times  the  computing  power  of  the 
old  SGI  Origin  2000  system  it  replaced. 

The  new  Origin  system  offers  a  number  of  im¬ 
provements  over  the  old  system.  The  processors 
on  the  new  03K  are  more  than  twice  as  fast  as  the 
processors  on  the  old  Origin  system,  with  400-MHz 
processors  as  compared  with  195  MHz.  This 
increase  in  processor  speed  resulted  in  the  ex¬ 
pected  two-times  speedup  for  most  of  the  codes 
tested  on  the  system.  A  few  codes  exhibited  even 
greater  speedup  that  was  attributed  to  general 
system  design  improvements.  The  new  system 
also  offers  two  times  the  per-processor  memory 
offered  by  the  old  system,  with  an  aggregate  total 
of  512  GB. 

The  new  03K  also  has  four  times  the  number  of 
processors  as  the  02K.  The  process  of  combining 
the  four  128-processor  images  occured  in  stages. 
In  this  first  stage,  the  four  128-processor  images 
were  combined  into  two  256-processor  images. 
This  increased  the  maximum  number  of  proces¬ 


sors  that  can  be  used  by  a  single  job  to  256.  In  the 
final  stage,  the  two  256-processor  images  were 
combined  into  a  single  512-processor  image, 
allowing  a  single  job  to  utilize  as  many  as 
512  processors. 

The  extra  processors,  increased  processor  speed, 
and  increased  memory  of  the  new  03K  system 
have  been  a  welcome  improvement.  Users  are 
now  able  to  run  larger  jobs,  utilize  more  proces¬ 
sors,  and  enjoy  a  faster  turnaround  time  on  their 
jobs.  With  these  improvements,  the  Origin 
problem  size  and  user  base  is  expected  to  increase 
significantly.  If  one  has  been  considering  a  move 
to  an  Origin  machine,  this  may  be  a  good  time  to 
make  the  transition.  To  obtain  additional  informa¬ 
tion  about  the  new  SGI  Origin  3800  system,  refer 
to  the  Web  site  at  www.wes.hpc.mil  or  contact  the 
Customer  Assistance  Center  at  1-800-500-4722. 
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technology  update 


technology  update 


Mass  Storage  Facility  Utiiities 

Drs.  Mark  R.  Fahey  and  Thomas  Oppe 

The  ERDC  MSRC  has  several  commands  for  manipulating  files  on  the  Mass  Storage  Facility  (MSF). 
These  commands  are  available  on  all  HPC  hosts,  as  are  man  pages  for  each  command. 

Note  that  there  are  two  “MSF”  machines:  new  and  old,  for  lack  of  better  naming  convention.  Whenever 
the  MSF  utilities  are  used  to  store  files,  the  files  are  stored  on  the  new  MSF.  However,  if  you  telnet  or  ftp 
to  msf,  you  will  be  logged  into  the  old  MSF. 

To  put  files  on  the  (new)  MSF,  the  commands  ms  f put  and  ms  f mput  are  available.  The  basic  usage  of 
these  two  commands  is  as  follows: 

msfput  local-file  [remote-file] 
msfmput  [-d  remote-path]  local-files 
Note  that  for  the  multiple  put,  the  remote  path  comes  first,  then  the  local  files  to  be  moved. 

Once  the  files  are  on  the  (new)  MSF,  the  files  can  be  manipulated  just  like  the  files  in  your  own 
homespace  using  MSF  utilities.  Commands  for  manipulating  files  already  on  the  MSF  are  as  follows: 
msfchmod  -m  mode  files,  where -m  mode  specifies  the  permissions  to  be  given 
msfls  [-o]  files 
msfmkdir  [-o]  remote-dir 
msfmv  [-o]  remote-file  target-file 
msfrm  [-o]  files 
msfrmdir  [-o]  remote-dir 
where  -o  forces  the  utility  to  perform  its  action  on  the  old  MSF. 

There  is  one  utility  that  checks  the  status  and  availability  of  the  MSF: 

msfstat  [-o] 

For  files  already  on  the  MSF  (new  or  old),  two  routines  are  used  to  retrieve  files: 

msf get  remote- file 

msfmget  [-d  path]  remote-files 

Note  that  just  the  like  the  msfmput,  the  remote  path  must  be  specified  first. 

There  are  also  two  commands  for  copying  files  from  the  old  MSF  to  the  new  MSF: 

msfcopy  remote-file 

msfmcopy  [-d  path]  remote-files 

See  the  man  pages  for  more  information  and  options  about  each  function. 


Origin  3000  versus  2000...A  Few  Tips 

Drs.  Mark  R.  Fahey  and  Thomas  Oppe 

The  ERDC  MSRC  has  installed  a  new  SGI  Origin  3800.  This  machine  replaced  the  Origin  2800.  Many 
users  had  grown  accustomed  to  the  Origin  2800  and  will  have  to  learn  a  few  new  things  to  run  on  the 
Origin  3800. 

New  qsub  option 

First,  when  submitting  batch  jobs  on  the  2800,  the  option  ncpus  was  used  to  specify  the  number  of 
processors.  On  the  3800,  this  option  is  not  used  nor  accepted  by  PBS.  The  correct  syntax  for  specifying 
the  number  of  CPUs  on  the  3800  is 

-lnodes=l :ppn=# 

where  #  is  the  desired  number  of  CPUs.  When  the  system  first  went  on-line,  the  maximum  value  for  ppn 
was  128.  The  maximum  possible  value  is  now  512.  Note  that  1  is  the  only  acceptable  numeric  value  for 
“nodes=l”. 
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SCS  Library 

Another  difference  many  users  will  encounter  is  the  name  of  the  SGI  scientific  library.  The  SGI  scientific 
library  was  well  known  to  be  named  COMPLIB  or  COMPLIB.SGIMATH.  However,  the  name  of  the  new 
scientific  library  is  SCSL. 

The  SGI  scientific  library  (SCSL)  is  a  collection  of  high-performance  scientific  libraries  that  provide 
support  for  mathematical  and  numerical  techniques  used  in  scientific  computing. 

To  get  more  information  about  the  SCSL  library,  type  “man  intro_libscsl”  or  “man 
intro_scsl”. 

A  very  important  difference  between  COMPLIB.SGIMATH  and  SCSL  is  that  COMPLIB  has  the  old 
LINPACK  and  EISPACK  routines  in  addition  to  the  LAPACK  routines,  but  SCSL  does  not.  Thus,  users 
will  have  to  either  port  their  LINPACK  and  EISPACK  calls  to  LAPACK  calls  or  obtain  the  EISPACK  and/ 
or  LINPACK  routines  they  need. 

The  following  sublibraries  are  included  in  the  current  release  of  SCSL: 

•  Signal  processing  routines 

-  Fast  Fourier  Transform  (FFT)  routines 

-  Convolution  routines 

-  Correlation  routines 

•  Linear  equation  solvers  for  sparse  systems  with  symmetric  nonzero  structure 

•  Vector- vector  linear  algebra  subprograms 

-  Level  1  Basic  Linear  Algebra  Subprograms  (Level  1  BLAS) 

•  Matrix- vector  linear  algebra  subprograms 

-  Level  2  Basic  Linear  Algebra  Subprograms  (Level  2  BLAS) 

•  Matrix-matrix  linear  algebra  subprograms 

-  Level  3  Basic  Linear  Algebra  Subprograms  (Level  3  BLAS) 

•  LAPACK,  a  public-domain  software  library  that  solves  problems  frequently  encountered  in  numerical 

linear  algebra 

SCSL  is  available  in  64-bit  and  32-bit  (n32)  modes  for  the  MIPS4  instruction  set  extensions.  To  use  the 
SCSL  routines,  load  the  library  with  the  -  Is cs  or  -lscs_mp  option.  The  -lscs_mp  option  directs 
the  linker  to  use  the  multiprocessor  version  of  the  library. 

When  linking  to  SCSL  with  -Iscs  or  -lscs_mp,  the  default  integer  size  is  4  bytes  (32  bits).  Another 
version  of  SCSL  is  available  in  which  integers  are  8  bytes  (64  bits).  This  version  allows  the  user  access  to 
larger  memory  sizes  and  helps  when  porting  legacy  Cray  codes.  It  can  be  loaded  by  using  the 
-lscs_i8  option  or  the  -  lscs_i8_mp  option.  A  program  may  use  only  one  of  the  two  versions; 
4-byte  integer  and  8-byte  integer  library  calls  cannot  be  mixed. 

Furthermore,  the  SCSL  library  has  new  FFT  routines.  The  following  table  shows  FFT  routines  from  the 
COMPLIB.SGIMATH  library  and  their  corresponding  routine  in  the  new  SCSL  library.  The  routines  in 
the  SCSL  library  can  be  used  to  do  both  initialization  and  FFT  computations. 


Complib 

SCSL 

cfftldi+cfftld 

-> 

ccf  f  t 

cf f tmldi+cf f  tmld 

— > 

ccf f tm 

cfft2di+cfft2d 

— > 

ccf f t2d 

cfft3di+cfft3d 

— > 

ccf f t3d 

Modules 

A  very  significant  change  from  the  2800  to  the  3800  configuration  is  the  use  of  modules.  Often  in  the 
course  of  tracking  down  a  computer  bug,  using  an  earlier  version  of  a  compiler  or  library  is  helpful.  The 
Modules  package  on  the  new  SGI  3800  platform  is  a  convenient  way  for  the  user  to  select  a  different 
version  of  software  without  going  to  the  trouble  of  modifying  his  PATH,  MANPATH,  and  other  environ¬ 
ment  variables  directly.  Modules  do  this  automatically. 
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First,  the  “module  list”  command  can  be  used  to  determine  what  versions  of  commonly  used  soft¬ 
ware  are  currently  being  used: 

ruby>  module  list 

Currently  Loaded  Module  files: 

1)  modules  3)  MPT.  1. 4.0.2 

2)  MIPSpro.7.3.1.1  4)  SCSL.l. 3.0.0 

The  “MIPSpro”  module  indicates  the  version  of  compilers  and  basic  libraries,  while  “MPT”  (for  “Mes¬ 
sage  Passing  Toolkit”)  indicates  the  MPI  and  PVM  message-passing  libraries,  and  “SCSL”  indicates  the 
SGI  scientific  library  containing  the  BLAS,  LAPACK,  and  other  routines.  To  determine  the  list  of  avail¬ 
able  modules  on  the  system,  type  “module  avail”: 

ruby>  module  avail 
-  /opt/modulef lies  - 

MIPSpro. 7. 2.1  MIPSpro. 7. 3. 1.1  MPT . 1 . 4  SCSL. 1.3. 0.0 

MIPSpro .7.3  MIPSpro . 7 . 3 . 1 . 2  MPT . 1 . 4 . 0 . 2 

Thus,  if  the  user  wishes  to  select  another  version  of  the  compiler,  the  “module  switch”  command  can  be 

used  to  select  an  earlier  version  of  the  compiler: 

module  switch  MIPSpro . 7 . 3 . 1 . 1  MIPSpro . 7 . 2 . 1 

This  switching  action  can  also  be  accomplished  in  two  steps  with 

module  unload  MIPSpro . 7 . 3 . 1 . 1 
module  load  MIPSpro . 7 . 2 . 1 

Note  that  the  term  “load”  does  not  indicate  that  a  particular  library  will  be  searched  automatically;  it 
simply  determines  which  version  of  the  library  will  be  searched.  The  user  must  still  use  “-Impi”, 
“-Iscs”,  “-lscs_mp”,  etc.,  on  the  linking  step  to  satisfy  external  references. 

To  determine  where  a  particular  library  is  located,  the  “module  display  <module_name>”  can  be 
used.  This  command  will  display  how  the  user’s  PATH  and  MANPATH  variables  are  altered  to  access  a 
particular  version: 

ruby>  module  display  SCSL. 1.3. 0.0 


/opt/modulef lies /SCSL .1.3 
Setenv  SCSL_SGI  to  /opt/ 
Prepend  /opt/scsl/1 . 3/usr 
Prepend  /opt/scsl/1 . 3/usr 
Prepend  /opt/scsl/1 . 3/usr 
Prepend  /opt/scsl/1 . 3/usr 
Prepend  /opt/scsl/1 . 3/usr 
Prepend  /opt/scsl/1 . 3/usr 
Prepend  /opt/MIPSpro/bin 
Append  /usr/share/catman : 
Prepend  /opt/scsl/1 . 3/usr 


.0.0: 

scsl/1 . 3 

/lib32  to  LD_LIBRARYN32_PATH 
/lib64  to  LD_LIBRARY64_PATH 
/Iib32/mips4  to  LD_LIBRARYN32_PATH 
/Iib64/mips4  to  LD_LIBRARY64_PATH 
/include  to  INCLUDE_PATH_SGI 
/bin  to  PATH 
to  PATH 

/usr/share/man : /usr/catman : /usr/man 
/share/catman  to  MANPATH 


to  MANPATH 


Finally,  if  the  user  forgets  a  “module”  keyword,  a  list  is  given  in  response  to  “module  help”,  “man 
module”  gives  the  man  page  for  the  modules  package.  The  most  commonly  used  commands  are  as 
follows: 

module  list  —  list  all  currently  loaded  modules 
module  avail  —  list  all  available  modules  on  the  system 

module  load  (or  add)  <module_name  (s)  >  —  load  one  or  more  modules 
module  unload  (or  rm)  <module_name  (s)  >  —  unload  one  or  more  modules 
module  switch  (or  swap)  <module_l>  <module_2>  —  replace  <module_l>  with 
<module_2> 

module  display  <module_name  ( s )  >  —  determine  how  one  or  more  modules  are 
accessed  through  changes  to  PATH  and  MANPATH 
module  help  —  lists  all  module  keywords 
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ERDC  MSRC  Scientific  Visuaiization  Center  Update 

Paul  Adams 


The  Scientific  Visualization  Center  (SVC)  team  works  with 
scientists  and  engineers  on  projects  to  merge  scientific  data 
with  computer  graphics  to  enable  researchers  to  perform 
visual  investigations  of  their  data.  These  visualizations 
enable  a  broader  understanding  of  the  technical  data  pre¬ 
sented.  Using  state-of-the-art  techniques  and  equipment, 
the  interdisciplinary  team  works  closely  with  a  wide  variety 
of  groups  to  provide  visual  clarification  on  various  studies. 

One  of  the  current  Challenge  Projects  on  the  ERDC 
MSRC  systems  is  the  AirBorne  Laser  II  (ABL)  project. 

The  problem  being  studied  is  that  of  turbulence  in  the 
atmosphere  distorting  a  laser  beam  as  it  travels  to  the 
target.  One  portion  of  the  ABL  II  project  deals  with  these 
distorting  effects  of  the  atmosphere  by  computationally 
modeling  and  simulating  earth’s  upper  troposphere  and 
lower  stratosphere.  Currently,  the  direct  numerical  simu¬ 
lation  of  stratified  shear  turbulence  and  gravity-wave 
breaking  runs  on  500  CPUs  on  the  ERDC  MSRC  T3E  and 
produces  over  400  GBytes  of  data  from  each  run.  Reduc¬ 
ing  that  data  to  more  understandable  information  is  the 
challenge  facing  the  ABL  project  and  the  SVC  team. 

Eigure  1  shows  the  viscous  dissipation  of  the  breakdown 
of  a  Kelvin-Helmoltz  (KH)  vortex  caused  by  stratified 
shear  flow.  Eigure  2  shows  thermal  dissipation  caused  by 
this  same  breakdown.  The  areas  of  greatest  dissipation, 
both  viscous  and  thermal,  are  colored  in  red.  The  areas  of 
zero  dissipation,  both  viscous  and  thermal,  are  colored  in 
black.  The  regions  with  the  largest  values  of  the  thermal 
dissipation  have  the  greatest  impact  on  the  optical  proper¬ 
ties  of  the  atmosphere  through  which  the  laser  beam  will 
travel  because  of  the  index  of  refraction’s  sensitivity  to 
thermal  gradients.  In  other  words,  it  is  in  these  regions 
that  the  high-power  laser  beam  will  be  distorted  and 
defocused  before  reaching  its  intended  target. 

Eigures  1  and  2  were  created  using  scientific  visualization 
package  OGLE  written  by  Dr.  Michael  Gourlay. 

A  second  Challenge  Project  at  the  SVC  is  called  the  Active 
Control  of  Euel  Injectors  in  Pull-Scale  Gas  Turbine 
Engines.  The  impetus  is  to  achieve  significant  improve¬ 
ment  in  the  performance  of  next-generation  gas  turbine 
engines  in  order  to  develop  lightweight,  compact,  fuel- 
efficient  propulsion  systems  for  helicopters  and  tanks  for 
the  Army.  A  reduction  in  fuel  consumption  would  be 
beneficial  by  extending  the  range  of  vehicles  as  well  as  by 
reducing  the  logistical  burden  of  transporting  fuel.  Experi¬ 
mentally  investigating  such  systems  is  difficult  because  of 
the  difficulty  in  obtaining  access  to  the  hot,  turbulent, 
reacting  region  in  the  combustor.  Eigure  3  shows  the 
flame  (in  yellow)  being  distorted  by  a  vortex  ring  (in  blue) 
in  the  combustion  chamber. 


Another  initiative  developed  by  the  SVC  team  was 
recently  shown  at  SuperComputing  2000  (SC2000). 
An  interactive  DVD  was  created  highlighting  the 
HPCMP.  Included  on  this  DVD  were  videos  from 
the  four  MSRCs,  as  well  as  information  from  the 
Distributed  Centers,  CHSSI  Projects,  and  Challenge 
Projects.  The  use  of  DVD  technology  is  anticipated 
to  play  a  significant  role  in  future  scientific 
visualization  capabilities. 


Figure  1.  Viscous  dissipation  of  the  breakdown  of  a  KH 
vortex  caused  by  stratified  shear  fiow 


Figure  2.  Thermai  dissipation  of  the  breakdown  of  a  KH 
vortex  caused  by  stratified  shear  fiow 


Figure  3.  Large-eddy  simuiation  of  a  swiriing  fiow  in  a 
combustor  showing  the  vortex  fiame  interaction 
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HPCMO  Benchmark 

Drs.  Dan  Duffy,  Mark  Fahey  and  William  A.  Ward,  Jr 

The  High  Performance  Computing  Modernization 
Office  (HPCMO),  which  guides  the  activities  of 
HPCMP,  organized  the  Benchmark  Working 
Group  to  oversee  the  construction  of  a  DoD  HPC 
benchmark.  This  benchmark  test  package  (BTP), 
as  currently  formulated,  has  three  components: 
synthetic  tests,  dedicated  application  tests,  and 
throughput  application  tests. 

Synthetic  benchmarks  are  programs  that  measure 
the  performance  of  a  computer  system  without 
doing  any  useful  work.  This  is  similar  in  concept 
to  the  way  that  a  person’s  heart  function  is  mea¬ 
sured  on  a  treadmill;  energy  is  expended,  cardiac 
performance  is  evaluated,  but  the  patient  does  not 
go  anywhere.  Constructed  by  Instrumental,  Inc., 
the  synthetic  tests  in  this  BTP  measure  peak  CPU, 
memory,  and  input/output  capabilities  of  candidate 
systems. 

If  synthetic  benchmarks  correspond  to  a  treadmill 
test,  then  application  benchmarks  are  like  walking 
to  the  store  to  bring  home  the  groceries;  a  mean¬ 
ingful  task  is  accomplished.  The  Computational 
Migration  Group  (CMG)  at  the  ERDC  MSRC  has 
assembled  these  tests  in  the  BTP.  Thirteen  actual 
application  codes  were  selected  for  inclusion  in 
the  BTP  based  on  MSRC  utilization  data.  They 
are  CHARGE,  COBALT-60,  CTH,  FEMD, 
FEMWATER123,  GAMESS,  ICEPIC,  LESlieSD, 
MD_MULTISCALE,  NLOM,  PRONTO, 
SARA-3D,  and  UNCLE.  These  codes  represent 


seven  of  the  ten  computational  technology  areas 
and  70  percent  of  all  node  hours  consumed  at  the 
DoD  MSRCs.  Four  of  the  codes,  COBALT-60 
(a  fluid  dynamics  code),  CTH  (a  structural 
mechanics  code),  GAMESS  (a  chemistry  and 
materials  code),  and  NLOM  (a  weather  and  ocean 
modeling  code),  alone  account  for  an  estimated 
32  percent  of  the  more  than  18  million  node  hours 
accrued  at  the  four  DoD  MSRCs  during  FY  2000. 
These  data  strongly  support  the  representativeness 
of  this  benchmarking  activity. 

Application  codes  are  used  in  two  types  of  tests 
within  the  BTP.  The  dedicated  application  tests 
involve  running  the  codes  for  various  inputs  and 
numbers  of  processors  on  an  otherwise  empty 
machine  in  order  to  determine  what  the  best 
possible  code  performance  might  be.  The 
throughput  application  tests  involve  running  a  mix 
of  the  codes  for  several  hours  in  order  to  gauge 
the  system’s  performance  in  an  operational  sense. 

The  benchmark  test  package  was  copied  onto  CDs 
and  made  available  to  vendors  in  November  2000. 
Hardware  vendors  received  copies  of  the  test 
package  and  returned  their  initial  results  to  the 
HPCMO  on  January  22,  2001.  These  results  are 
currently  being  used  by  the  HPCMO  to  provide 
guidance  for  future  system  procurements.  The 
HPCMO  also  views  production  of  this  BTP  as  an 
initial  step  in  a  long-term  performance  metrics 
effort  within  the  DOD  HPC  community. 
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Dr.  Terry  Disz,  Argonne  National  Laboratory 
briefs  ERDC  MSRC  personnel  on  Access  Grid 


Argonne  National  Laboratory 


On  January  9,  2000,  Dr.  Terry  Disz,  Ms.  Lisa  Childers,  and  Mr.  Robert  Olson 
from  Argonne  National  Laboratory  (ANL)  and  Mr.  Tony  Rimovsky  from  the 
National  Center  for  Supercomputing  Applications  visited  the  ERDC  MSRC  to 
brief  and  exchange  information  with  the  MSRC  personnel  on  the  Access  Grid 
multimedia  conferencing  system.  This  system,  which  was  designed  by  ANL, 
provides  multigroup  collaboration  across  the  Internet.  The  ERDC  MSRC  is 
installing  this  system  and  will  use  it  to 
provide  collaboration  with  users  located 
throughout  the  DoD  and  university 
community. 


Leading-Edge  Research 

A  partnership  of  Government,  academic,  and  industry  collaborators, 
namely,  the  ERDC  MSRC,  the  Arctic  Region  Supercomputing  Center  at 
the  University  of  Alaska  Eairbanks,  SGI,  and  CSC,  to  test  an  advanced 
supercomputing  system  is  ongoing.  The  project  was  initiated  with  the 
installation  of  the  Origin  3800  at  ERDC.  A  partnership  like  this  enhances 
information  exchange  opportunities  and  offers  scientists  the  computational 
capability  necessary  to  conduct  leading-edge  research. 


Mississippi  chaps  Stephen 
Jones  and  Brad  Comes  visit 
Alaska  in  September  2000 


Virginia  Bedford, 

Technical  Services 

Director,  and  Kurt  Carlson,  Systems  Analyst,  both  from  the  Arctic  Region  Supercomputing  Center,  are  shown  in  front 
of  the  Origin  3800  at  the  ERDC  MSRC  with  Jay  Cliburn,  Director  of  System  Integration  and  Technology,  ERDC  MSRC 
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Center  of  Higher  Learning 

On  November  20,  2000,  members  from  the  Center  of  Higher  Learning  (CHL)  visited  the  ERDC  MSRC. 
The  CHL  is  a  consortium  of  three  Mississippi  universities  and  one  community  college  located  at  the  John  C. 
Stennis  Space  Center:  The  University  of  Southern  Mississippi,  Mississippi  State  University,  The  Univer¬ 
sity  of  New  Orleans,  and  Pearl  River  Community  College. 


Members  from  the  Center  of  Higher  Learning  are  shown  with  Brad  Comes  in  the  ERDC 
MSRC  scientific  visuaiization  center.  Shown  around  tabie  (ieft  to  right)  are  Dr.  Adel  All, 
Visualization,  University  of  Southern  Mississippi;  Dr.  Walt  Smith,  Litton;  Brad  Comes, 
ERDC  MSRC  Director;  Dr.  Conrad  Johnson,  University  of  Southern  Mississippi; 
and  Dr.  Peter  Ranelli,  Technical  Director,  Center  of  Higher  Learning 


In  attendance  from  CHL 
were  Dr.  Peter  Renault, 

Dr.  Walt  Smith,  Dr.  Adel 
Ali,  and  Dr.  Conrad 
Johnson.  The  visitors 
toured  the  supercomputing 
and  scientific  visualization 
centers.  A  possible  col¬ 
laborative  effort  between 
the  two  sites  was  discussed, 
including  remote, 
immersive  visualization. 


High  Performance  Computing  Modernization  Office 

The  Director  of  the  High  Performance  Computing  Modernization  Program  (HPCMP),  Cray  Henry,  along 
with  Dr.  Frank  Mello,  Chief  Scientist,  HPCMP;  Dr.  Larry  Davis,  Deputy  Director,  HPCMP;  and  John 
Baird,  Project  Manager,  Shared  Resource  Centers,  HPCMP,  came  to  the  ERDC  Headquarters  in 
Vicksburg,  Mississippi,  on  October  31  -  November  1,  2000,  to  visit  with  the  ERDC  Director,  Dr.  James 
Houston,  some  of  the  ERDC  HPC  users  in  the  Structures  and  Coastal  and  Hydraulics  laboratories,  and  to 

participate  in 
meetings  at  the 
ERDC  MSRC 
regarding 
benchmarking, 
system  perfor¬ 
mance  metrics, 
and  Challenge 
Project  support. 


HPCMO  visitors  tour  the  ERDC  Information  Technology  Laboratory  (ITL).  Pictured  (left  to  right) 
John  Baird,  Project  Manager,  Shared  Resource  Centers,  HPCMP;  Brad  Comes,  ERDC  MSRC 
Director;  Cray  Henry,  HPCMP  Director;  Timothy  Abies,  ITL  Acting  Director;  Dr.  Larry  Davis, 
Deputy  Director,  HPCMP;  and  Dr.  Frank  Mello,  Chief  Scientist,  HPCMP 


/ 
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User  Requirements — HPCMP 

Tom  Crimmins,  Army  Research  Laboratory  MSRC,  Dr.  Larry  Davis,  Jann  Ensweiler,  and  Cathy  McDonald, 
HPCMP,  and  Dr.  Danny  Weddle,  Naval  Air  Warfare  Center  Aircraft  Division,  visited  the  ERDC  MSRC 
on  September  11-12,  2000,  to  meet  with  the  ERDC  HPC  users  and  coordinate  their  requirements  and 
HPC  activities. 


Dr.  Robert  Jensen,  ERDC  Coastal  and  Hydraulics  Laboratory, 
discusses  his  HPC  projects  with  the  HPCMP  team  members 


HPCMP  team  with  ERDC  MSRC  team  in  the  ERDC  supercomputing  facility  Shown  (left 
to  right)  Dr  Danny  Weddle,  Naval  Air  Warfare  Center  Aircraft  Division;  Dr  Larry  Davis, 
HPCMP;  Dennis  Gilman  and  David  Stinson,  ERDC  MSRC;  Jann  Ensweiler,  HPCMP; 
Dean  Hampton,  ERDC  MSRC;  Cathy  McDonald,  HPMCP; 
and  Tom  Crimmins,  Army  Research  Laboratory  MSRC 
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Continued  from  page  5 


LTG  Robert  B.  Flowers,  Commanding 
General,  US.  Army  Corps  of  Engineers, 
January  23,  2001 


CRT  Jay  Ferreira,  ERDC;  LTC  Mike  Kavanagh,  Australian 
Liaison  Officer,  Fort  Leonard  Wood,  Missouri; 
Wayne  Jones,  ERDC  ITL;  and  Brad  Comes, 

ERDC  MSRC  Director 


(Left  to  right)  Dr.  Richard  Olsen,  ERDC  Geotechnical  and  Structures 
Laboratory  (GSL);  Glen  Browning,  ERDC  MSRC;  Dr.  Lawson  Smith, 
GSL;  Mario  Castaneda,  SERNA;  Paul  Adams  and  Tom  Biddiecome, 
ERDC  MSRC;  Raul  Flores,  GeoConsult;  and 
Mike  Valladares,  U.S.  Army  Corps  of  Engineers — 
Tegucigalpa — Honduras,  November  30,  2000 


Todd  Wilson,  U.S.  Army  Engineer  District,  Omaha;  Harry  Hendier, 
Earth  Tech  Laboratory,  Austin,  Texas;  Tyler  Bowiey,  ERDC 
Environmental  Laboratory;  and  Dennis  Gilman,  ERDC  MSRC, 
December  12,  2000 


U.S.  Army  Aviation  and  Missile  Command  representatives. 
Research  Development  and  Engineering  Center, 
Huntsville,  Alabama,  and  David  Stinson  (far  right), 
ERDC  MSRC,  October  25,  2000 


MG  Milton  Hunter  (center).  Deputy  Commanding 
General,  Military  Programs,  U.S.  Army  Corps 
of  Engineers,  January  23,  2001 
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COL  James  S.  Weller  (right  front),  ERDC 
Commander,  November  7,  2000 


(Left  to  right)  David  Stinson  and  MAJ  Robert  Sheldon, 
ERDC,  and  COL  William  Bailey,  British  Liaison  Officer, 
Fort  Leonard  Wood,  Missouri,  September  6,  2000 


(Left  to  right)  COL  James  S.  Weller,  ERDC  Commander; 
Timothy  Abies,  Acting  Director,  ERDC  Information 
Technology  Laboratory  (ITL);  and  Deborah  Dent, 
Special  Assistant  to  the  ITL  Acting  Director 


/ - \ 

The  ERDC  MSRC  welcomes  comments  and  suggestions  regarding  The  Resource  and  invites 
article  submissions.  Please  send  submissions  to  the  following  e-mail  address: 

info-hpc@wes.hpc.mil 

\ _ _ _ / 

ERDC  MSRC  Technical  Reports 

00-31  “Effect  of  the  Wave  Propagation  Scheme  on  Nearshore  Wave  Predictions,”  Stephen  Womom,  David  J.S. 
Welsh,  and  Keith  W.  Bedford. 

00-32  “Practical  Experiences  with  the  Eortran  Pthreads  API,”  Clay  P.  Breshears  and  Phu  Luong. 

00-33  “Comparison  of  OpenMP  and  Pthreads  within  a  Coastal  Ocean  Circulation  Model  Code,”  Clay  P. 

Breshears,  and  Phu  Luong. 

00-34  “Tools  for  Understanding  Program  Performance,”  John  Mellor-Crummey. 

00-35  “Using  OpenMP  and  Threaded  Libraries  to  Parallelize  Scientific  Applications,”  Daniel  Duffy. 

00-36  “Analysis  of  HPC  Usage:  ERDC  MSRC,”  Daniel  Duffy. 

00-37  “On  Coupling  the  SWAN  and  WAM  Wave  Models  for  Accurate  Nearshore  Wave  Predictions,”  Stephen 
Wornom,  David  Welsh,  and  Keith  Bedford. 

00-38  “Effect  of  the  Wave  Propagation  Scheme  on  SWAN  Nearshore  Wave  Predictions,”  Stephen  Wornom, 

David  Welsh,  and  Keith  Bedford. 

01-01  “Emulating  Co-Array  Eortran  with  Pthreads,”  Richard  Hanson  and  Stephen  Wornom. 

01-02  “2000  ERDC  MSRC  PET  Training  Activities,”  Wayne  Mastin. 

01-03  “PET  Core  Support  and  Eocused  Efforts  for  the  period  27  March  2001  through  30  September  2001.” 

01-04  “EORTRAN  77  to  EORTRAN  90  Source  Code  Conversion  and  Maintenance  Tools,”  Richard  Weed. 

These  and  other  technical  reports  can  be  accessed  at 
WWW.  wes.  hpc.  mil 


The  Resource,  Spring  2001 


ERDC  MSRC 


35 


visitors  technical  reports 


training  acronyms 


Below  is  a  list  of  acronyms  commonly  used  among  the  DoD  HPC  community.  You  will  find  these  acro¬ 
nyms  throughout  the  articles  in  this  newsletter. 


ADPC 

Automated  Data  Processing  Center 

AFOSR 

Air  Eorce  Office  of  Scientific 

Research 

AFRL 

Air  Force  Research  Laboratory 

AFSC 

Air  Force  Systems  Command 

ANL 

Argonne  National  Laboratory 

ASC 

Aeronautical  Systems  Center 

BTP 

Benchmark  Test  Package 

CFD 

Computational  Fluid  Dynamics 

CHL 

Center  of  Higher  Learning 

CHSSI 

Common  High  Performance  Com¬ 
puting  Software  Support  Initiative 

CMG 

Computational  Migration  Group 

COE 

Corps  of  Engineers 

CPUs 

Central  Processor  Units 

CSC 

Computer  Sciences  Corporation 

CSM 

Computational  Structural  Mechanics 

CWO 

Climate,  Weather,  and  Ocean 
Modeling  and  Simulation 

DNS 

Domain  Name  System 

DoD 

Department  of  Defense 

DoE 

Department  of  Energy 

DOT&E 

Director  of  Operational  Test  and 
Evaluation 

EQM 

Environmental  Quality  Modeling 
and  Simulation 

ERDC 

Engineer  Research  and  Development 
Center 

EDDI 

Fiber  Distributed  Data  Interface 

EFT 

Fast  Fourier  Transform 

HBCU/MI 

Historically  Black  Universities  and 
Colleges/Minority  Institutions 

HPC 

High-Performance  Computing 

HPCMO 

High  Performance  Computing  Moderni¬ 
zation  Office 

HPCMP 

High  Performance  Computing  Moderni¬ 
zation  Program 

IMN 

Internal  Major  Shared  Resource  Center 
Network 

ITL 

Information  Technology  Laboratory 

LFT&E 

Live  Fire  Test  and  Evaluation 

MCWG 

Metacomputing  Working  Group 

MPI 

Message-Passing  Interface 

MSF 

Mass  Storage  Facility 

MSRC 

Major  Shared  Resource  Center 

NASA 

National  Aeronautics  and  Space  Admin¬ 
istration 

NAVO 

Naval  Oceanographic  Office 

PET 

Programming  Environment  and  Training 

POM 

Princeton  Ocean  Model 

SCSL 

SGI  Scientific  Library 

SMP 

Symmetric  Multiprocessor 

SVC 

Scientific  Visualization  Center 

TCP/IP 

Transport  Control  Protocol/Internet 
Protocol 

WES 

Waterways  Experiment  Station 

WMD 

Weapons  of  Mass  Destruction 

The  contents  of  this  newsletter  are  not  to  be  used  for  advertising,  publication,  or  promotional  purposes.  Citation  of 
trade  names  does  not  constitute  an  official  endorsement  or  approval  or  the  use  of  such  commercial  products. 


Training  Schedule* 

June  5-7 

HLA  Regional  Training  Event  (DMSO) 

July 

Computing  Portals  and  the  Grid  Computing 
Environment  (ESU) 

July  31-August  2 

Advanced  MPI  Tips  and  Tricks  (Tennessee) 

September  11-12 

Advanced  UNIX  (OSC) 


*Additional  courses  may  be  offered.  Please  check  the  ERDC  MSRC  Web  page  at 

www.wes.hpc.mil 
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